Compact eye-tracked head-mounted display

ABSTRACT

Eye-tracked head-mounted displays are provide which, in one aspect, may utilize the same optics for eyetracking and image viewing, with a selected portion of the optics used for an eyetracking optical path and a selected portion of the display optics used for an image viewing optical path.

RELATED APPLICATIONS

This is a continuation application of U.S. application Ser. No.15/446,134, filed Mar. 3, 2017, which is a continuation application ofU.S. application Ser. No. 14/372,292, filed Jul. 15, 2014, which is a371 application of International Application No. PCT/US2013/022918 filedJan. 24, 2013, which claims the benefit of priority of U.S. ProvisionalApplication No. 61/632,441, filed on Jan. 24, 2012 and claims thebenefit of priority of U.S. Provisional Application No. 61/687,607,filed on Apr. 25, 2012 and claims the benefit of priority of U.S.Provisional Application No. 61/699,493, filed on Sep. 11, 2012, theentire contents of which applications are incorporated herein byreference.

GOVERNMENT LICENSE RIGHTS

This invention was made with government support under Grant No.IIS1115489 awarded by the National Science Foundation. The governmenthas certain rights in the invention.

FIELD OF THE INVENTION

The present invention relates generally to eye-tracked head-mounteddisplays, and more particularly, but not exclusively, to eye-trackedhead-mounted displays which may utilize the same optics for eyetrackingand image viewing, with a selected portion of the optics used for aneyetracking optical path and a selected portion of the display opticsused for an image viewing optical path.

BACKGROUND OF THE INVENTION

Head-mounted display (HMD) technologies have been applied to a widerange of scientific and engineering domains. Examples of applicationsinclude flight simulation, scientific visualization, medicine,engineering design, education and training, wearable computing, andentertainment systems. In the domain of augmented reality, HMDs are oneof the enabling technologies for merging virtual views with physicalscenes, which may enable a physician to see a 3D rendering of theanatomical structures or CT images of a patient superimposed onto thepatient's anatomy, such as the abdomen, for example. In the domain ofwearable computing, an HMD creates a mobile display solution that offersmuch more attractive image quality and screen size than other popularmobile platforms such as smart phones and PDAs. In the foreseeablefuture, such mobile displays may appear as elegant as a pair ofsunglasses and may become an integral part of many people's dailyactivities to retrieve information and connect with people instantly.

In parallel with HMD technologies, various eyetracking technologies havebeen developed and applied to several disciplines including visionresearch, human computer interfaces, tele-operation environments, andvisual communication. The benefits of eyetracking for multi-modalhuman-computer interfaces and the technical benefits of data compressionhave been well-recognized and studied. For instance, multi-resolutiongaze-contingent display and image processing schemes have been proposedto effectively save data transmission bandwidth in communication, andimprove rendering speed of 3D scenes using foveated level-of-detailmanagement methods, and to achieve wide FOV high-resolution display andimaging systems.

The concept of creating an integrated eyetracked HMD (ET-HMD) system hasbeen explored in various levels. An ET-HMD is able to display monocularor stereoscopic virtual images as a classical HMD does, whileadditionally tracking the gaze direction of the user. A fully-integratedET-HMD offers multi-fold benefits, not only to fundamental scientificresearch but also to emerging applications of such technology. Forinstance, many research efforts are concerned about how human usersperceive and organize spatial information, interact with suchinformation, and navigate within 3D virtual spaces. Eyetrackingcapability in HMDs adds a very valuable tool and objective metric forscientists to quantitatively assess user interaction with 3Denvironments and investigate the effectiveness of various 3Dvisualization technologies for various specific tasks includingtraining, education, and augmented cognition tasks. From the technologypoint of view, eyetracking capability integrated with HMD systems can beutilized to improve size and depth perception accuracy in stereoscopicdisplays. Eyetracking capability may help to create solutions to theFOV-resolution tradeoff through a fovea-contingent display scheme and tothe accommodation-convergence contradiction by using vari-focal planedisplay methodology. From the application point of view, an ET-HMDoffers unique opportunities for novel interactive interfaces for peoplewith proprioceptive disabilities where eye gaze instead of hands or feetcan be used as a method of interaction and communication.

Despite significant advancements and commercial availability ofstand-alone HMD and eyetracking technologies, integrating these twostand-alone technologies imposes significant challenges in creating acompact, portable, accurate and robust system. Although severalpioneering efforts were made to develop ET-HMD technologies and tooptimize these two technologies in a systematic approach, none of theexisting technological solutions offers a truly portable, lightweight,and robust system that conforms to the form factor of an eyeglass-styledisplay. For many demanding applications, lightweight and compactnessare critical. For instance, to support Amyotrophic Lateral Sclerosis(ALS) patient communication, the integrated system has to be lightweightso that the patients are able to bear the weight with theirsignificantly weakened muscles and very limited mobility.

Over the past decades, many different optical design approaches havebeen applied to HMD designs to improve the system performance. Thesemethods include applying catadioptric technique, introducing newelements such as aspherical surfaces, using holographic and diffractiveoptical components, exploring new design principles such as usingprojection optics to replace an eyepiece or microscope type lens systemin a conventional HMD design, and introducing tilt and decenter or evenfreeform surfaces. Few of these optical design methods are capable ofcreating a wide field-of-view, compact, and lightweight HMD that isnonintrusive and can be considered as being eyeglass-style near-eyedisplays. Integrating eyetracking capability to these technologies isvery challenging and adds significant weight, volume, and complexity.

Adding eyetracking capability to HMDs started as early as the highresolution inset displays by CAE Corporation. This pioneering work wasnot intended for mobile compact ET-HMD systems. Also, others used amechanical driving device to move a high resolution inset in abench-prototype stereoscopic display. ISCAN Corporation worked tointegrate an ISCAN eyetracker into a V8-HMD from Virtual ResearchCorporation to study software-based fovea-contingent display scheme.This method of integrating commercially available HMDs and eye-trackersis referred to as the functionality integration approach, in which twoseparate instruments are brought together at a later stage ofutilization. Though the functionality integration approach has theadvantage of being a simple solution with low development cost, itgenerally does not take advantage of low-level optimization and lacksthe attributes of compactness, accuracy, and robustness.

In contrast to the functionality integration approach, a systematicapproach, where the system is conceived and optimized as one singleinstrument from a fundamental design perspective, has many advantages increating a fully integrated ET-HMD instrument. The significant benefitsof the systematic approach include the ability to explore the designconstraints and requirements for both the display and eyetracker units,conceive new solutions, and optimize the designs for a compact androbust system. Pioneering efforts have been made to explore thepossibility of a complete integration with low-level optimization.Following these earlier efforts, Hua and Rolland collaboratively pursueda fully integrated design approach, developed robust eyetracking methodsand algorithms for an ET-HMD system, and designed an optical see-throughET-HMD optical system based on the concept of head-mounted projectiondisplays. FIG. 1 shows the first-order layout of the ET-HMD opticalsystem, in which the optical system was simplified with ideal lensmodules to emphasize the concept and the scale. (Curatu, C., Hong Hua,and J. P. Rolland, “Projection-based head-mounted display witheye-tracking capabilities,” Proceedings of the SPIE InternationalSociety for Optical Engineering, Vol. 5875, San Diego, USA, August 2005.Curatu, C., J. P. Rolland, and Hong Hua, “Dual purpose lens for aneye-tracked projection head-mounted display,” Proceedings ofInternational Optical Design Conference, Vancouver, Canada, June 2006.).The design took a full integration approach and combined most of theoptical paths for the display and eyetracking subsystems. The sameprojection optics was shared for both display and eye imaging functions.The main limitation of this design, however, was that the overall volumeof the integrated ET-HMD system, although significantly improved overothers, was still bulky and heavy.

The key challenges of creating a truly portable, lightweight, compactET-HMD solution lies in addressing two cornerstone issues: (1) anoptical method that enables the design of an HMD system with an elegantform factor as compelling as a pair of sunglasses, which has been apersistent dream for both technology and application developers; and (2)an optical method that allows the integration of the eyetrackingcapability without adding significant weight and volume to the system.

SUMMARY OF THE INVENTION

An ET-HMD system using a video-based feature tracking method typicallyrequires at least three unique optical paths: an illumination path, aneye imaging path, and a virtual display path. Through the illuminationpath the eye is illuminated by typically near infrared light-emittingdiodes (NIR LEDs) to create imaging features such as darkened orbrightened pupil and/or Purkinje features for tracking. Through theimaging path, an eye image with the tracking features is captured forfeature detection and tracking. Through the display path, a virtualimage displayed on a miniature display device is created througheyepiece optics for information viewing. One of the innovations of thepresent invention is an optical scheme that can uniquely combine thesethree optical paths through the same core optics, which may be aneyepiece, projection lens, or other optics structure.

For example, in one of its aspects, the present invention may usefreeform optical technology along with an innovative optical scheme thatcan uniquely combine eye imaging optics for eyetracking with the displayoptics for information viewing. (Thus, as used herein in connection withdescription of the present invention, the terms “display optics” and“imaging optics” may refer to the same physical optics, which physicaloptics may also be called the “core optics”.) Optionally, the eyeillumination optics may also be combined. As such, in one of itsadvantages the present invention avoids the limitation imposed by priorapproaches where the optical systems for the HMD and eyetracking pathsare treated separately, and where rotationally symmetric opticalsurfaces are mostly used. However, though possibly more limiting, theoptical scheme of integrating eyetracking with HMD disclosed in thepresent invention is not limited to freeform optics. The core optics forthe ET-HMD system in accordance with the present invention can beapplied to conventional HMD optics.

In an exemplary configuration, the present invention may provide aneye-tracked head-mounted display comprising a micro-display forgenerating an image to be viewed by a user; the micro-display may have adisplay optical path and an exit pupil associated therewith. A firstplane may be located at the micro-display and a second plane located atthe exit pupil. An image sensor may be configured to receive reflectedoptical radiation from the second plane reflected from a user's eye, andmay have a sensor optical path associated therewith. In addition, theeye-tracked head-mounted display may include display optics disposed inoptical communication with the micro-display along the display opticalpath and in optical communication with the image sensor along the sensoroptical path. The display optics may include a selected surface closestto the micro-display and the image sensor and be located relative to themicro-display and image sensor such that the display and image sensoroptical paths impinge upon differing respective portions of the selectedsurface. The display and image sensor optical paths may partiallyoverlap at the selected surface. The display and image sensor opticalpaths may each comprise respective optical axes at the display opticsand image sensor, respectively, which axes may be coaxial or tiltedrelative to one another. In addition, the eye-tracked head-mounteddisplay may include a stop at the first plane, where the stop has atleast one aperture therein disposed at a location along the sensoroptical path. Likewise, the eye-tracked head-mounted display may includea stop having at least one aperture therein disposed at a location alongthe sensor optical path between the sensor and selected surface. Ineither configuration, the stop or aperture may include a pin-hole likeaperture. In one exemplary configuration, the display optics may includea freeform optical element, a rotationally symmetric optical element,and/or a freeform optical prism. The display optics may include anaspheric surface.

In addition, the eye-tracked head-mounted display may include anillumination source for generating optical radiation to illuminate thesecond plane to effect illumination of the user's eye. The displayoptics may be configured to collimate the optical radiation from theillumination source. The illumination source may be located in the firstplane or at a different location, such as off axis from the optical axisof the display optics.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing summary and the following detailed description ofexemplary embodiments of the present invention may be further understoodwhen read in conjunction with the appended drawings, in which:

FIG. 1 schematically illustrates a conventional eyetracked head-mounteddisplay (ET-HMD) system based on rotationally symmetric opticaltechnology;

FIGS. 2A, 2B schematically illustrate images from two different IRillumination strategies, with FIG. 2A showing an eye image of a brighteye pupil and four glints resulting from an on-axis illuminationstrategy where four NIR LEDs are arranged nearly co-axially with theoptical axis of the eye imaging optics, and FIG. 2B showing an eye imageof a dark eye pupil and four glints resulting from an off-axisillumination strategy where the four NIR LEDs are placed away from theoptical axis of the eye imaging optics;

FIG. 3A schematically illustrates an exemplary optical system inaccordance with the present invention shown as a monocular opticalmodule;

FIG. 3B schematically illustrates an exemplary system in accordance withthe present invention of illumination units and eye imaging unitsdisposed around a microdisplay panel;

FIG. 4 schematically illustrates a block diagram of an exemplary systembased on freeform prism technology in accordance with the presentinvention shown as a monocular optical module;

FIGS. 5A-5D schematically illustrate an exemplary design of an opticalsee-through HMD in accordance with the present invention, with FIG. 5Ashowing the eye illumination and imaging paths, FIG. 5B showing thevirtual display path, FIG. 5C showing a freeform prism shared by eyeillumination, eye imaging, and virtual display paths, and FIG. 5Dshowing a freeform auxiliary lens attached to the freeform prism, whichenables see-through capability;

FIG. 6 schematically illustrates an optical layout and raytracing of anexemplary optimized ET-HMD system in accordance with the presentinvention using the 2-reflection freeform prism structure of FIG. 5D;

FIG. 7 schematically illustrates a 3D model of an exemplary ET-HMDoptical system in accordance with the present invention;

FIG. 8 schematically illustrates a model of an exemplary binocularET-HMD prototype in accordance with the present invention based on theoptical design in FIGS. 6 and 7;

FIGS. 9A-9D illustrate the polychromatic modulation transfer function(MTF) of 20 sampled fields across the field of view in the HMD virtualdisplay path with a 4-mm centered pupil of the design of FIG. 6;

FIG. 10 illustrates the distortion grid across the field of view in theHMD virtual display path of the design of FIG. 6;

FIG. 11 illustrates the modulation transfer function of sampled fieldsacross the field of view in the eye imaging path of the design of FIG.6;

FIG. 12 illustrates the distortion grid across the field of view in theeye imaging path of the design of FIG. 6;

FIGS. 13A-13D illustrate the polychromatic modulation transfer function(MTF) of 20 sampled fields across the central field of view of 30×22degrees in the HMD see-through path with a 4-mm centered pupil of thedesign of FIG. 6;

FIG. 14 illustrates the distortion grid across the field of view in theHMD see-through path of the design of FIG. 6;

FIGS. 15A, 15B illustrate an exemplary design of the optical schemeshown in FIG. 3 in accordance with the present invention; and

FIG. 16 schematically illustrates an exemplary implementation of theoptical scheme shown in FIG. 3 in accordance with the present inventionbased on rotationally symmetric optics.

DETAILED DESCRIPTION OF THE INVENTION

Referring now to the figures, wherein like elements are numbered alikethroughout, FIG. 3A schematically illustrates an exemplary system layout300 in accordance with the present invention for achieving a compactET-HMD system. In this exemplary layout 300, the same core optics 310may serve the functions of eye imaging, display viewing, and/or eyeillumination. This simplification stems from an insightful observationon the unique conjugate planes in the eye illumination path 305, eyeimaging path 307, and display path 309. In addition, differing portionsalong the clear aperture of the core optics 310 may be used for the eyeillumination path 305, eye imaging path 307, and display path 309. Forinstance, at a selected surface of the core optics 310 located closestto the micro-display, two or more of the eye illumination path 305, eyeimaging path 307, and display path 309 (e.g. eye imaging path 307 anddisplay path 309) can impinge upon differing respective portions of theselected surface, though partial overlap is permitted.

In the display path 309, the core optics 310, which in this contextfunctions as display optics, forms a magnified virtual image of themicrodisplay 320 seen by the eye 10. The microdisplay unit 320 can beany type of self-emissive, or illuminated pixel arrays that can serve asan image source, including, but not limited to, a liquid crystal onsilicon (LCoS) display device, a liquid crystal display (LCD) panel, anorganic light emitting display (OLED), ferroelectric liquid crystal onsilicon (FLCoS) device, digital mirror device (DMD), or amicro-projector built upon these aforementioned or other types ofmicro-display devices, and additional optional optics may be providedbetween the microdisplay 320 and core optics 310, as desired orrequired. The magnified virtual image, which may appear to be at aninfinite or finite distance from the eye 10, corresponds to theconjugate focal plane of the microdisplay 320. The eye pupil 12 may beco-located with the exit pupil 312 of the display path 309. The chiefrays of the display through the center of the pupil 12 (shown in solidlines in FIG. 3A) define the field height on the microdisplay 320, andthus they are separable on the microdisplay surface. In the eyeillumination path 305, one or multiple NIR LEDs (near-infraredlight-emitting diodes) 330 may be mounted around the microdisplay 320 toilluminate the eye through the display/core optics 310, FIG. 3B. Thedisplay/core optics 310 may collimate the LED light and create auniformly illuminated area on the eye area through multiple virtual LEDsources created through the display/core optics 310. Such an off-axisillumination arrangement can create a dark-pupil effect and formmultiple glint images of the NIR LEDs 330 through the reflection off theanterior cornea.

In the eye imaging path 307, the eye pupil 12 becomes the object thatneeds to be imaged. A stop 340 may be placed around the microdisplay320. Considering the pupil-field relationship of the microdisplay 320and the eye pupil 12 described earlier, the chief rays of differentobject fields in the display path become the marginal rays of theon-axis object point in the eye imaging path 307, and thus all the raysthrough the same point on the eye pupil 12 will be imaged onto the samepoint on the IR imaging sensor 360. These rays, however, intersect withthe microdisplay surface at unique locations. Therefore, in the imagingpath 307, a stop 340 is properly designed and placed around themicrodisplay 320 such that it does not affect the display path 309 andyet is sufficient to collect rays to form eye images in the eye imagingpath 307. In the illustration shown in FIG. 3B, the stop 340 may beprovided in the form of pin-hole like small apertures 350 or may be aselected area surrounding the microdisplay 320. A separate image sensor360 may be associated with each pin-hole like aperture 350.

As one of its benefits, the optical layout 300 for combining two orthree unique optical functions has applicability to virtually all typesof optical structures suitable for HMD optics. For instance, anexemplary configuration with a conventional eyepiece optics based onrotationally symmetric optical elements has been designed, as discussedbelow in connection with FIG. 16.

As to the eyetracking function aspect specifically, several differenteyetracking techniques exist that may be used to monitor eye movements,which fall into three categories: electro-oclography, scleral searchcoil, and various video-based feature tracking approaches. Among thesemethods, video-based feature tracking, which detects and tracks featuresin captured eye images, can be the least intrusive and most convenientapproach to track eye movement.

Under near infrared NIR illumination, the eye images 201, 202 typicallyhave two types of features that can be readily identified and measured,FIGS. 2A, 2B. One feature is known as the first Purkinje image, or glint6, which refers to the reflection image of a point light source formedby the anterior surface of the cornea, FIG. 2B. The second feature isthe eye pupil 12. FIGS. 2A-2B demonstrate examples of IR-illuminated eyeimages 201, 202. Depending on configuration of the IR illuminators,e.g., NIR LEDs 330, an on-axis illumination strategy where the IRilluminators are arranged nearly co-axial with the optical axis of theeye imaging optics leads to a bright pupil 2, FIG. 2A, while an off-axisillumination strategy where the IR illuminators are placed away from theoptical axis of the eye imaging optics leads to a darkened pupil 4 withglint(s) 6, FIG. 2B. The pupil and glint features may then be utilizedfor eye movement tracking.

Among the video-based feature tracking methods, the pupil-cornealreflection tracking method, which relates the eye movements with thevector difference between the pupil center and the glint center, may bea most suitable approach in an ET-HMD system. In this method, one ormultiple NIR light emitting diodes (NIR LED), e.g., NIR LEDs 330, may beused to illuminate the eye 10, and the illuminated eye 10 may thenimaged by the imaging sensor 360, such as an infrared CCD. The eye pupil12, the first Purkinje image (or glint), and/or the iris 11 may betracked simultaneously or separately. Each NIR LED 330 may form a glint6 or a first Purkinje image. The pupil 12 and first Purkinje featuresmove proportionally with eye rotation and differentially between eachother. The differential vector between the two features may be used todetermine the point-of-regard of the eye 10. To some extent this methodcan tolerate helmet slippage in a HMD system, which causes orientationchange of the imaging sensor 360 relative to the eye 10 and confuses theeye movements.

In another of its significant aspects, the present invention may utilizefreeform optical technology in the core optics 310 to achieve anultra-compact and lightweight ET-HMD with see-through capability. FIG. 4shows a block diagram 400 of an exemplary approach to a compacteyetracked HMD design in accordance with the present invention based onfreeform optical technology. In one exemplary implementation, awedge-shaped freeform prism 410 or waveguide-type freeform prism may beused in the core optics 310, which allows the ray paths to be foldedwithin a multi-surface prism structure and helps reduce the overallvolume and weight of the display optics when compared with designs usingrotationally symmetric elements. Applying freeform optical technologyenables full integration of the functions of HMD optics and eyetrackinginto a compact form. The freeform prism 410 may be made of moldableplastic for lightweight and low cost.

In this approach, the freeform prism 410 may serve two or more uniqueoptical functions. First, the freeform prism 410 may serve as the coreelement in the eye imaging path 407 that captures NIR-illuminated eyeimages 401 of a user and tracks eye movements using the captured eyeimages 401. Unlike a conventional imaging system, which typicallyemploys rotationally symmetrical optical surfaces in the lensconstruction and typically requires the imaging lenses remain collinearwith the detector 460 and the objects to be captured, the freeform prism410 folds the light path within a single element so that the imagedetector 460 may be placed on the side of the freeform prism 410.Second, the same freeform prism 410 may serve as display viewing opticsfor viewing images on the microdisplay 420 in the display path 409.Third, the prism 410 may serve as the core element in the illuminationpath 305 that collimates the light from one or multiple of the NIR LEDs430. Alternatively, the NIR LEDs may illuminate the eye area directlywithout passing through the prism 410 (or core optics 310). In eithercase, the NIR LEDs 430 may uniformly and non-invasively illuminate theeye area and form critical features (e.g. glints 6 and darkened pupil 4)that are to be imaged for eyetracking. Finally, if an opticalsee-through ET-HMD system is required for applications where a directview of the real world is critical, the prism 410 may be cemented with afreeform corrective lens 415. The freeform corrector 415 can correct theviewing axis deviation and undesirable aberrations introduced by theprism 410 and enables see-through capability of the system 400 whichoffers low peripheral obscurations and minimized distortions to thereal-world view 411. Overall, the unique optical scheme of the presentinvention can enable the combination of the optical paths for the eyeimaging 407 and the virtual display 409, and optionally eye illumination405, through the same freeform prism 410 and can achieve thecapabilities of eyetracking and display with minimum hardware cost.

Example 1

A first exemplary configuration 500 in accordance with the presentinvention utilizes wedge-shaped freeform prism 510 with two reflections,FIGS. 5A-5D. In this embodiment, the freeform prism 510 may serve asmany as three core functions: (1) as an illumination optic thatcollimates the light from one or multiple NIR LEDs 530 to uniformly andnon-invasively illuminate the eye area to be imaged; (2) as the coreelement of an eye imaging optic that captures NIR-illuminated eye imagesto enable eye movement tracking; and (3) as an eyepiece optic of an HMDsystem to view images on a microdisplay 520. These three unique opticalpaths may be combined by the same freeform prism 510 to achieve thecapabilities of eyetracking and display. Additionally, the same prism510 when cemented with a freeform corrective lens enables thesee-through capability of an optical see-through HMD system.Alternatively, freeform prism 510 may omit the core function as anillumination optic.

The wedge-shaped freeform prism 510 may include three optical surfaces,at least of one of which may be an aspheric surface with or withoutrotational symmetry. One innovation of the present invention is theoptical approach that can uniquely combine the two or three uniqueoptical paths (i.e., two or more of the eye illumination path 505, eyeimaging path 507, and display path 509) via the single freeform prism510. FIG. 5A shows the schematic design of the eye illumination andimaging optics, which includes freeform prism 510. In the illuminationpath 505, a ray emitted from an NIR LED 530 is first refracted by thesurface 3, followed by two consecutive reflections by the surfaces 1′and 2, and finally is transmitted through the surface 1 and reaches theeye 10. The reflection on surface 1′ may satisfy the condition of totalinternal reflection (TIR). The light emitted by the LEDs 530 may becollimated by the prism 510, yielding a uniform illumination to the eye10. The NIR illuminated eye 10 may then be imaged by an IR image sensor560. In the eye imaging path 507, light rays scattered off the eye 10may be first refracted by the surface 1, followed by two consecutivereflections by the surface 2 and 1′, and finally may be transmittedthrough the surface 3 and reach the sensor 560. Additional lenses 562may be inserted between the surface 3 of the prism 510 and the imagesensor 560 to improve optical performance of the eye imaging. Asmall-aperture stop 550 may be placed near or inside the lenses 562 toconfine the light received by the imaging sensor 560.

FIG. 5B schematically illustrates the display path 509 of HMD opticsusing the freeform prism 510 to magnify the image on a microdisplay 520,forming a virtual image at a comfortable viewing distance. A ray emittedfrom a point on the microdisplay 520 may be first refracted by thesurface 3 of the freeform prism 510, followed by two consecutivereflections by the surfaces 1′ and 2, and finally may be transmittedthrough the surface 1 to reach the exit pupil 512 of the system 500. Thereflection on surface 1′ may satisfy the TIR condition. Rather thanrequiring multiple elements, the optical path is naturally folded withinthe prism structure. Additional lenses may be inserted between thesurface 3 of the prism 510 and the microdisplay 520 to further improveoptical performance of the display path 509.

FIG. 5C schematically illustrates the integrated system 500 where theillumination, imaging and display optics comprise the same prism 510 andthe illumination LEDs 530 and a pinhole-like stop 550 are placed aroundthe edge 540 of the microdisplay 520 to form a high-quality eye image.One example of the stop and LED configurations is illustrated in FIG.5C. It is worth noting the stop 550 and LEDs 530 may be placed in otherlocations at the periphery around in the microdisplay 520. In addition,the stop 550 and LEDs 530 may or may not be co-planar with themicrodisplay 520. Additional lenses may be used in one or more of theillumination path 505, eye imaging path 507, and display path 509 toimprove the system performance. Moreover, at the surface closest to themicrodisplay 520, surface 3, the illumination path 505, eye imaging path507, and display path 509 may impinge upon differing respective portionsof surface 3 (though partial overlap is permitted).

To enable see-through capability, the surface 2 of the prism 510 may becoated as a half mirror. The rays from the microdisplay 520 may bereflected by the surface 2 while the rays from a real-world scene aretransmitted. FIG. 5D schematically illustrates a freeform auxiliary lens515, consisting of two freeform surfaces 4 and 5, cemented with theprism 510 to correct the viewing axis deviation and aberrationsintroduced by the freeform prism 510 to the real world view path 511.The surface 4 of the auxiliary lens 515 usually has the sameprescription as the surface 2 of the prism 510 and the surface 5 of theauxiliary lens 515 is optimized to correct the axis deviation and theaberrations. The auxiliary lens 515 does not noticeably increase thefootprint or weight of the overall system. Overall, the exemplary system500 provides a lightweight, compact, robust, and eyetracked HMD solutionwith a less obtrusive form factor than any existing HMD approaches canpotentially deliver, which is further demonstrated by computer analysisof the design.

FIG. 6 schematically illustrates the two-dimensional optical layout ofan optimized system based on the 2-reflection wedge-shaped freeformprism 510 depicted in FIG. 5. In this implementation, an imaging lens562 may be used to improve the performance of the eye imaging path 507.The stop 550 may be positioned close to the surface 3 of the prism 510.The NIR-LED(s) 530 may be positioned around the microdisplay 520. FIG. 7schematically illustrates a 3D model 700 of the exemplary optical systemof FIG. 5D, and FIG. 8 schematically illustrates the 3D model of abinocular ET-HMD prototype 800 based on the optical design shown inFIGS. 6 and 7. The specifications of the overall system are listed inTable 1.

TABLE 1 Optical System Specifications Parameter Values Virtual displaysystem Display FOV 46° (Diagonal), 40° (Horizontal) × 22° (Vertical)Exit pupil diameter 10 mm (zero vignette), offer an eyebox of 18 mm fora 4 mm pupil. Eye clearance 19 mm Display resolution 1920 × 1200 colorpixels Distortion <8% across FOV Image quality (MTF) Average 20% at 50lps/mm and average 30% at 35 lps/mm Design wavelength 450-650 nmSee-through viewing optics See-through FOV Approximately 100°(Diagonal), 80° (Horizontal) × 50° (Vertical) Distortion <10% at theedge and less than 2% at the center Image quality (MTF) >50% at 0.5cycles/min and greater than 0.3 at 1 cycles/min Design wavelength450-650 nm Eye tracking sub-system FOV ( Imaged eye area) 30 mm (H) × 20mm (V) Image quality (MTF) Average 10% at 50 lps/mm and average 25% at30 lps/mm Distortion <5% across the imaged area Design wavelength750~900 nm

An exemplary optical prescription of the freeform prism 510 is listed inthe Tables 2-4 for surfaces 1, 2, and 3, respectively. Of the threeoptical surfaces in the prism 510, the surface 1 is an anamorphicaspheric surface (AAS). The sag of an AAS surface is defined by

$z = {\frac{{c_{x}x^{2}} + {c_{y}y^{2}}}{1 + \sqrt{1 - {\left( {1 + K_{x}} \right)c_{x}^{2}x^{2}} - {\left( {1 + K_{y}} \right)c_{y}^{2}y^{2}}}} + {{AR}\left\{ {{\left( {1 - {AP}} \right)x^{2}} + {\left( {1 + {AP}} \right)y^{2}}} \right\}^{2}\left. \quad{{{BR}\left\{ {{\left( {1 - {BP}} \right)x^{2}} + {\left( {1 + {BP}} \right)y^{2}}} \right\}^{3}} + {{CR}\left\{ {1 - {CP}} \right)x^{2}} + {\left( {1 + {CP}} \right)y^{2}}} \right\}^{4}} + {\quad{{{DR}\left\{ {{\left( {1 - {DP}} \right)x^{2}} + {\left( {1 + {DP}} \right)y^{2}}} \right\}^{5}},}}}$where z is the sag of the free-form surface measured along the z-axis ofa local x, y, z coordinate system, c_(x) and c_(y) are the vertexcurvature in x and y axes, respectively, K_(x) and K_(y) are the conicconstant in x and y axes, respectively, AR, BR, CR and DR are therotationally symmetric portion of the 4th, 6th, 8th, and 10th orderdeformation from the conic, AP, BP, CP, and DP are the non-rotationallysymmetric components of the 4th, 6th, 8th, and 10th order deformationfrom the conic.

Surface 2 of the prism 510 may be an XY polynomial surface defined by:

${z = {\frac{c\left( {x^{2} + y^{2}} \right)}{1 + \sqrt{1 - {\left( {1 + k} \right){c^{2}\left( {x^{2} + y^{2}} \right)}}}} + {\sum\limits_{j = 1}^{66}{C_{j}x^{m}y^{n}}}}},{j = {{\left\lbrack {\left( {m + n} \right)^{2} + m + {3n}} \right\rbrack/2} + 1}}$where z is the sag of the free-form surface measured along the z-axis ofa local x, y, z coordinate system, c is the vertex curvature (CUY), k isthe conic constant, and Cj is the coefficient for x^(m)y^(n).

Surface 3 may be an aspheric surface with a rotationally symmetrickinoform diffractive optical element, with the sag of the asphericsurface defined by:

${z = {\frac{{cr}^{2}}{1 + \sqrt{1 - {\left( {1 + K} \right)c^{2}r^{2}}}} + {Ar}^{4} + {Br}^{6} + {Cr}^{8} + {Dr}^{10} + {Er}^{12} + {Fr}^{14} + {Gr}^{16} + {Hr}^{18} + {Jr}^{20}}},$where z is the sag of the surface measured along the z-axis of a localx, y, z coordinate system, c is the vertex curvature, k is the conicconstant, A through J are the 4th, 6th, 8th, 10th, 12th, 14th, 16th,18th, and 20th order deformation coefficients, respectively.

TABLE 2 Optical surface prescription of surface 1 of the freeform prismX Curvature (c_(x))  −1.348215E−02 Y Curvature (c_(y))  2.004523E−03 YConic Constant (K_(Y))  0.998125E+01 4th Order Symmetric Coefficient(AR) −3.9067945E−06 6th Order Symmetric Coefficient (BR) −9.5768964E−178th Order Symmetric Coefficient (CR) −2.8799927E−15 10th Order SymmetricCoefficient (DR) −8.7077963E−16 X Conic Constant (K_(X)) −1.5687534E+014th Order Asymmetric Coefficient (AP) −3.2949463E−01 6th OrderAsymmetric Coefficient (BP) −2.0405356E+02 8th Order AsymmetricCoefficient (CP) −8.0782710E+00 10th Order Asymmetric Coefficient (DP)−2.72019184E−01 

TABLE 3 Optical surface prescription of surface 2 of the freeform prism510 Y Curvature −1.26056882299E−02 X**3 * Y**4 (SCO X3Y4 | C33)0.0000000000E+00 Y Radius −7.93292664201E+01 X**2 * Y**5 (SCO S2Y5 |C34) 2.0693559836E−10 Conic Constant (SCO K | C1)  1.99429650209E+00 X *Y**6 (SCO XY6 | C35) 0.0000000000E+00 X (SCO X | C2)  0.00000000000E+00Y**7 (SCO Y7 | C36) 2.1203645386E−10 Y (SCO Y | C3)  0.00000000000E+00X**8 (SCO X8 | C37) 2.6638311623E−12 X**2 (SCO X2 | C4)−2.8963611697E−03 X**7 * Y (SCO X7Y | C38) 0.0000000000E+00 X * Y (SCOXY | C5) 0.00000000000E+00 X**6 * Y**2 (SCO X6Y2 | C39) 4.2552541871E−12Y**2 (SCO Y2 | C6) 5.13151841830E−04 X**5 * Y**3 (SCO X5Y3 | C40)0.0000000000E+00 X**3 (SCO Y3 | C7) 0.00000000000E+00 X**4 * Y**4 (SCOX4Y4 | C41) −4.101261981E−12 X**2 * Y (SCO X2Y | C8) −1.6871196613E−05X**3 * Y**5 (SCO X3Y5 | C42) 0.0000000000E+00 X Y**2 (SCO XY2 | C9)0.00000000000E+00 X**2 * Y**6 (SCO X2Y6 | C43) 3.9696325158E−12 Y**3(SCO Y3 | C10) −3.9628025988E−05 X * Y**7 (SCO XY7 | C44)0.0000000000E+00 X**4 (SCO X4 | C11) 5.63763951591E−07 Y**8 (SCO Y8 |C45) 1.7421792489E−11 X**3 * Y (SCO X3Y | C12) 0.00000000000E+00X**9(SCO X9 | C46) 0.0000000000E+00 X**2 * Y**2 (SCO X2Y2 | C13)−5.1451820404E−07 X**8 * Y (SCO X8Y | C47) 2.8416565461E−13 X * Y**3(SCO XY3 | C14) 0.00000000000E+00 X**7 * Y**2 (SCO X7Y2 | C48)0.0000000000E+00 Y**4 (SCO Y4 | C15) 1.52902584933E−06 X**6 * Y**3 (SCOX6Y3 | C49) 7.7200373777E−13 X**5 (SCO X5 | C16) 0.00000000000E+00X**5 * Y**4 (SCO X5Y4 | C50) 0.0000000000E+00 X**4 * Y (SCO X4Y | C17)2.30036831137E−08 X**4 * Y**5 (SCO X4Y5 | C51) −6.188783932E−13 X**3 *Y**2 (SCO X3Y2 | C18) 0.00000000000E+00 X**3 * Y**6 (SCO X3Y6 | C52)0.0000000000E+00 X**2 * Y**3 (SCO X2Y3 | C19) 3.82949206634E−08 X**2 *Y**7 (SCO X2Y7 | C53) 1.7935251959E−14 X * Y**4 (SCO XY4 | C20)0.00000000000E+00 X * Y**8 (SCO XY8 | C54) 0.0000000000E+00 Y**5 (SCO Y5| C21) −9.3057372440E−08 Y**9 (SCO Y9 | C55) −1.391093985E−13 X**6 (SCOX6 | C22) −2.3473886032E−09 X**10 (SCO X10 | C56) −2.6923251198E−15X**5 * Y (SCO X5Y | C23) 0.00000000000E+00 X**9 * Y (SCO X9Y | C57)0.00000000000E+00  X**4 * Y**2 (SCO X4Y2 | C24) −2.4682522624E−09 X**8 *Y**2 (SCO X8Y2 | C58) −1.5546422781E−14  X**3 * Y**3 (SCO X3Y3 | C25)0.00000000000E+00 X**7 * Y**3 (SCO X7Y3 | C59) 0.00000000000E+00  X**2 *Y**4 (SCO X2Y4 | C26) −3.5764311583E−09 X**6 * Y**4 (SCO X6Y4 | C60)−1.0384073178E−14  X * Y**5 (SCO XY5 | C27) 0.00000000000E+00 X**5 *Y**5 (SCO X5Y5 | C61) 0.0000000000E+00 Y**6 (SCO Y6 | C28)−4.3636504848E−09 X**4 * Y**6 (SCO X4Y6 | C62) 3.8750232363E−14 X**7(SCO X7 | C29) 0.00000000000E+00 X**3 * Y**7 (SCO X3Y7 | C63)0.0000000000E+00 X**6 * Y (SCO X6Y | C30) −1.8300632292E−10 X**2 * Y**8(SCO X2Y8 | C64) −3.094245370E−14 X**5 * Y**2 (SCO X5Y2 | C31)0.00000000000E+00 X * Y**9 (SCO XY9 | C65)  0.000000000E+00 X**4 * Y**3(SCO X4Y3 | C32) −1.0237987168E−10 Y**10 (SCO Y10 | C66) −3.15607172E−14

TABLE 4 Optical surface prescription of surface 3 of the freeform prism510 Y Radius −1.5000000000E+01 Conic Constant (K) −8.1715030467E+00 4thOrder Coefficient (A) −3.5999478362E−05 6th Order Coefficient (B) 4.1811989405E−07 8th Order Coefficient (C) −2.0382499300E−09 10th OrderCoefficient (D)  3.7498678418E−12 Diffraction Order 1 ConstructionWavelength (nm) 550 R**2 (HCO C1) −3.2332326174E−03 R**4 (HCO C2) 4.1482610496E−05 R**6 (HCO C3) −4.2185152895E−07 R**8 (HCO C4) 1.8253428127E−09 R**10 (HCO C5) −2.7615741244E−12

An exemplary optical prescription of surface 5 of the freeform corrector515 lens is listed in Table 5. Surface 4 of the lens 515 has the sameprescription as the surface 2 of the prism 510 and the surface 5 of thelens 515 is an XY polynomial surface defined by the same equation as forsurface 2.

TABLE 5 Optical surface prescription of surface 5 of the freeformcorrector lens Y Curvature −4.9680519947E−03 X**3 * Y**4 (SCO X3Y4 |C33) 0.000000000E+00 Y Radius −2.0836485397E+02 X**2 * Y**5 (SCO S2Y5 |C34) −1.546473120E−11  Conic Constant (SCO K | C1) 9.64085149870E+00 X *Y**6 (SCO XY6 | C35) 0.000000000E+00 X (SCO X | C2) 0.00000000000E+00Y**7 (SCO Y7 | C36) −2.36018874E−11 Y (SCO Y | C3) 0.00000000000E+00X**8 (SCO X8 | C37) −1.08111832E−12 X**2 (SCO X2 | C4) −3.7131327715E−03X**7 * Y (SCO X7Y |C38)  0.00000000E+00 X * Y (SCO XY | C5)0.00000000000E+00 X**6 * Y**2 (SCO X6Y2 | C39)  −9.9791583E−13 Y**2 (SCOY2 | C6) 3.49505772747E−03 X**5 * Y**3 (SCO X5Y3 | C40)  0.0000000E+00X**3 (SCO Y3 | C7) 0.00000000000E+00 X**4 * Y**4 (SCO X4Y4 | C41) −8.6526761E−12 X**2 * Y (SCO X2Y | C8) −1.5261510919E−07 X**3 * Y**5(SCO X3Y5 | C42)  0.00000000E+00 X Y**2 (SCO XY2 | C9)  0.0000000000E+00X**2 * Y**6 (SCO X2Y6 | C43)  −3.9166253E−12 Y**3 (SCO Y3 | C10) −9.571153875E−08 X * Y**7 (SCO XY7 | C44)  0.00000000E+00 X**4 (SCO X4| C11)  −1.871425121E−07 Y**8 (SCO Y8 | C45)  1.45724979E−11 X**3 * Y(SCO X3Y | C12)  0.000000000E+00 X**9 (SCO X9 | C46)  0.00000000E+00X**2 * Y**2 (SCO X2Y2 | C13)  −2.91567230E−06 X**8 * Y (SCO X8Y | C47) 3.51280116E−15 X * Y**3 (SCO XY3 | C14)  0.000000000E+00 X**7 * Y**2(SCO X7Y2 | C48)  0.00000000E+00 Y**4 (SCO Y4 | C15)  −8.129645853E−07X**6 * Y**3 (SCO X6Y3 | C49)  6.69288844E−15 X**5 (SCO X5 | C16) 0.0000000000E+00 X**5 * Y**4 (SCO X5Y4 | C50)  0.00000000E+00 X**4 * Y(SCO X4Y | C17)  1.4913830346E−09 X**4 * Y**5 (SCO X4Y5 | C51) 6.15758388E−14 X**3 * Y**2 (SCO X3Y2 | C18)  0.0000000000E+00 X**3 *Y**6 (SCO X3Y6 | C52)  0.00000000E+00 X**2 * Y**3 (SCO X2Y3 | C19) 2.4358316954E−09 X**2 * Y**7 (SCO X2Y7 | C53)  1.94985620E−14 X * Y**4(SCO XY4 | C20)  0.0000000000E+00 X * Y**8 (SCO XY8 | C54) 0.00000000E+00 Y**5 (SCO Y5 | C21)  4.1849942311E−09 Y**9 (SCO Y9 |C55)  4.24428256E−14 X**6 (SCO X6 | C22)  −9.610954967E−10 X**10 (SCOX10 | C56)  9.43112860E−16 X**5 * Y (SCO X5Y | C23)  0.0000000000E+00X**9 * Y (SCO X9Y | C57)  0.00000000E+00 X**4 * Y**2 (SCO X4Y2 | C24)5.6221328063E−10 X**8 * Y**2 (SCO X8Y2 | C58)  2.10137145E−15 X**3 *Y**3 (SCO X3Y3 | C25)  0.0000000000E+00 X**7 * Y**3 (SCO X7Y3 | C59) 0.00000000E+00 X**2 * Y**4 (SCO X2Y4 | C26)  7.656820595E−10 X**6 *Y**4 (SCO X6Y4 | C60) 1.130922231E−14 X * Y**5 (SCO XY5 | C27) 0.000000000E+00 X**5 * Y**5 (SCO X5Y5 | C61) 0.000000000E+00 Y**6 (SCOY6 | C28)  −2.99368733E−09 X**4 * Y**6 (SCO X4Y6 | C62) −1.93900784E−15X**7 (SCO X7 | C29)   0.00000000E+00 X**3 * Y**7 (SCO X3Y7 | C63)0.000000000E+00 X**6 * Y (SCO X6Y | C30)   −4.2039898E−12 X**2 * Y**8(SCO X2Y8 | C64) 7.080929646E−15 X**5 * Y**2 (SCO X5Y2 | C31)   0.0000000E+00 X * Y**9 (SCO XY9 | C65) 0.000000000E+00 X**4 * Y**3(SCO X4Y3 | C32)   −7.665313E−12 Y**10 (SCO Y10 | C66) −1.96970504E−14

On the display side of the exemplary design, the prism 510 provides adiagonal FOV of 46 degrees, or 40 degrees horizontally and 22 degreesvertically. It supports a microdisplay 520 with a pixel size of ˜8 μmand a diagonal size of 0.9″ or smaller. In the prototype that wasfabricated, a 0.86″ microdisplay with an aspect ratio of 16:9 and aresolution of 1920×1200 pixels was used.

The exemplary design achieves high image contrast and resolution. FIGS.9A-9D illustrate the polychromatic modulation transfer function (MTF) of20 sampled fields across the field of view in the HMD path with a 4-mmcentered pupil. The MTF curves demonstrate an average contrast of 0.2 atthe cutoff resolution of 50 lps/mm (equivalent to a 10 μm pixelresolution) and an average contrast greater than 0.3 at the cutoffresolution of 35 lps/mm (equivalent of approximately 15-um pixelresolution). FIG. 10 further demonstrates the distortion grid of thevirtual display path.

On the eye imaging and illumination side, one or more NIR LEDs 530 areplaced around the image source to create a uniformly illuminated eyearea through the freeform prism 510. The freeform prism 510 is able toprovide uniform illumination for an eye area of approximately 30 mm×20mm in the horizontal and vertical directions, respectively. The sameilluminated eye area is captured by a high resolution NIR sensor 560.The imaged area is sufficient to allow eye movement tracking. Theresolvable pixel size of the eye imaging path is about ˜10 um. FIG. 11illustrates the modulation transfer function (MTF) of the eye imagingpath. The MTF curves demonstrate an average contrast of 0.1 at thecutoff resolution of 50 lps/mm (equivalent to a 10 μm pixel resolution)and an average contrast greater than 0.25 at the cutoff resolution of 30lps/mm (equivalent of approximately 16-um pixel resolution). FIG. 12further illustrates the distortion grid of the eye imaging path.

On the see-through side of the system 500, the cemented prism 510 andfreeform corrective lens 515 provide a diagonal FOV of approximately 100degrees, or 80 degrees horizontally and 50 degrees vertically. Thesee-through FOV is designed to be much larger than the virtual displayFOV for improved situational awareness. The eyebox size of thesee-through system is optimized to be larger than the virtual displaysystem to further improve ease of use and viewing comfort. This designembodiment achieves high image contrast and resolution. FIGS. 13A-13Dillustrate the polychromatic modulation transfer function (MTF) of 20sampled fields across the center 30×22 degrees of field of view insee-through path with a 4-mm centered pupil. The MTF curves demonstratenearly diffraction limited performance. In FIGS. 13A-13D, 0.5 cycles/mincorresponds to 1 minute of arc spatial resolution, which is theresolvability of 20/20 vision, and 1 cycles/min corresponds to 0.5minute of arc spatial resolution, which is the resolvability of 20/15vision. The average MTF across the sampled fields is greater than 0.5 atthe cutoff resolution of 0.5 cycles/min (equivalent to 1 minute of arcangular resolution) and an average contrast greater than 0.4 at thecutoff resolution of 1 cycles/min (equivalent to 0.5 minutes of arcangular resolution). The average MTF across the entire 80×50 see-throughFOV is greater than 0.35 at the cutoff frequency of 0.5 cycles/min. FIG.14 further illustrates the distortion grid of the see-through displaypath across the entire FOV. The distortion in the central 40×22 degreesis less than 2% and the distortion across the whole field is less than8%.

Example 2

FIGS. 15A-15B schematically illustrate an exemplary design of a secondconfiguration of the present invention, where the stop 1540 of theimaging system 1500 may surround the microdisplay 1520. The microdisplayplane is divided into three regions: an IR-transmissive area 1527 thatallows collecting the rays by an IR sensor 1560 and which may serve asthe stop 1540 for eye imaging on IR sensor 1560; the active area of themicrodisplay 1520 (non-transmissive) corresponding to the active displayarea which blocks the IR rays from reaching the IR sensor 1560; and, athird non-transmissive frame 1523 between the IR transmissive andmicrodisplay areas corresponding to a physical frame of the microdisplaywhich also blocks the rays from reaching the IR sensor 1560. In theimaging system 1500 the respective optical axes of the prism 1510,microdisplay 1520, and IR sensor 1560 may be coaxial. As such, the IRsensor 1560 may be placed after the microdisplay 1520 to capture theimage of the eye pupil. The distance from the IR sensor 1560 to theprism 1510 depends on the image location of the eye pupil through thefreeform prism 1510, which ultimately depends on the design of thedisplay path. For instance, if the freeform prism 1510 is designed to betelecentric or close to telecentric in the display space, the chief rayswill be nearly parallel to each other and perpendicular to themicrodisplay surface before they intersect with the microdisplay 1520.This means the image of the eye pupil through the prism 1510 is locatedat infinity or a significantly far distance. In this case, one or moreadditional imaging lenses 1562 may need to be inserted between the IRsensor 1560 and the prism 1510 to reduce the overall length of the eyeimaging path and achieve good image quality, FIG. 15A.

On the other hand, if the freeform prism 1510 is designed to benon-telecentric (i.e., the chief rays will converge to a point at someshort distance behind the prism 1510), the eye pupil is imaged at afairly close distance by the prism 1510 and the IR sensor 1560 can beplaced directly behind the prism 1510 without the need for additionalimaging lenses 1562. In practice, the condition of telecentricity ornear-telecentricity is often desirable when designing the display pathbecause the virtual image appears to be more uniform across the entireFOV. This condition may be required when the microdisplay 1520 onlyemits or reflects light within a narrow angle (e.g. devices such as LCoStype microdisplays). When the microdisplay 1520 offers a wide emissionangle (e.g. OLED), the telecentricity condition can be relaxed.

The NIR LEDs may be placed around the stop 1540 in the similar way asdescribed in FIG. 3B, or alternatively the NIR LEDs 1530 may be placedaround the edge of the prism 1510 and directly illuminate the eye 10 asshown in FIG. 15A. Moreover, the NIR LEDs 1530 around the edge of theprism 1510 to directly illuminate the eye 10 without use of the prism1530 may be implemented in any other configuration of the invention,including those depicted in FIGS. 5A-6 or 16, for example.

Example 3

FIG. 16 schematically illustrates an exemplary design 1600 of theoptical scheme shown in FIG. 3 utilizing on rotationally symmetricoptics for the core optics 310. Instead of using a compact freeformbased prism 510, a four-element viewing optics 1610 is used as the coreoptics 310 for display viewing, eye imaging and eye illumination. Themicrodisplay 1620 may be placed at the focal plane of the optics 1610.One light source 1630 (such as an NIR-LED) may be placed around theimage source 1620. A pinhole-like stop 1640 and micro-imaging lens 1662may also be placed around the edge of the image source 1620 to form eyeimages on the imaging sensor 1660. Additional light sources 1630 andimaging subsystems (micro-imaging lens 1662 and imaging sensors 1660)can be arranged around the image source 1620 as needed for differentapplications. In the exemplary design 1600 the respective optical axesof the display optics 1610 and microdisplay 1620 may be coaxial, whilethe respective optical axes of one or more of the image sensor 1660,light source 1630, and microdisplay 1620 may be tilted and/or decenteredrelative to one another. As with the freeform configurations, at thesurface closest to the microdisplay 1620, surface 8, the illuminationpath, eye imaging path, and display path impinge upon differingrespective portions of surface 8, though partial overlap is permitted,e.g., as illustrated between the imaging path and display path.

The viewing optics 1610 can provide a diagonal FOV of 40 degrees, 20-mmeye-relief and 10-mm eye-pupil size, and can support an image source1620 with a diagonal size of 0.8″ or smaller. One or more NIR LEDs 1630may be placed around the microdisplay 1620 to create a uniformlyilluminated eye area through the viewing optics. The viewing optics 1610is able to provide uniform illumination for an eye area of approximately15 mm×15 mm. The same illuminated eye area may be captured by a highresolution NIR sensor 1630. The imaged area is sufficient to allow eyemovement tracking.

An exemplary optical prescription of the design 1600 is provided inTables 6-9.

TABLE 6 Optical surface prescription of the viewing optics 1610 SURFACESURFACE RADIUS THICKNESS NUMBER TYPE (MM) (MM) MATERIAL OBJECT INFINITYINFINITY AIR 1 (STOP) 0 20 AIR 2 SPHERE 38.747568 13 ACRYLIC 3 SPHERE−68.038477 2.940552 AIR 4 SPHERE 87.660626 4.795025 ACRYLIC 5 SPHERE−52.591345 0.1 AIR 6 SPHERE 29.845125 10.782261 NBK7 7 SPHERE −23.0167988 SF61 8 SPHERE 30.000017 7.076910 AIR 9 (MICRO- INFINITY 0 DISPLAY)

TABLE 7 Optical surface prescription of the imaging lens 1662 SURFACESURFACE RADIUS THICKNESS MATER- NUMBER TYPE (MM) (MM) IAL 10 INFINITYINFINITY AIR (STOP) 11 ASPHERE 41.495014 3.183189 ACRYLIC 12 ASPHERE−2.858167 5.988505 AIR 13 INFINITY 0 (IR SENSOR) DECENTER COORDINATES OFSURFACE 10 (STOP) RELATIVE TO SURFACE 8 Y DECENTER (MM) Z DECENTER XTILT (ADE) 8.7401084 3 8.3216381

Surfaces 11 and 12 may be aspheric surfaces with the sag of the asphericsurface defined by:

${z = {{\frac{{cr}^{2}}{1 + \sqrt{1 - {\left( {1 + K} \right)c^{2}r^{2}}}}{Ar}^{4}} + {Br}^{6} + {Cr}^{8} + {Dr}^{10} + {Er}^{12} + {Fr}^{14} + {Gr}^{16} + {Hr}^{18} + {Jr}^{20}}},$where z is the sag of the surface measured along the z-axis of a localx, y, z coordinate system, c is the vertex curvature, k is the conicconstant, A through J are the 4th, 6th, 8th, 10th, 12th, 14th, 16th,18th, and 20th order deformation coefficients, respectively.

TABLE 8 Optical surface prescription of surface 11 of the imaging lens YRadius 41.495014 Conic Constant (K) −20 4th Order Coefficient (A)−1.021763E−02 6th Order Coefficient (B) −6.885433E−04 8th OrderCoefficient (C) −3.263238E−04 10th Order Coefficient (D) 0

TABLE 9 Optical surface prescription of surface 12 of the imaging lens YRadius −2.858167 Conic Constant (K) −1.750218 4th Order Coefficient (A)−7.851177E−03 6th Order Coefficient (B) −1.064232E−04 8th OrderCoefficient (C) −4.912295E−05 10th Order Coefficient (D) 0

These and other advantages of the present invention will be apparent tothose skilled in the art from the foregoing specification. Accordingly,it will be recognized by those skilled in the art that changes ormodifications may be made to the above-described embodiments withoutdeparting from the broad inventive concepts of the invention. It shouldtherefore be understood that this invention is not limited to theparticular embodiments described herein, but is intended to include allchanges and modifications that are within the scope and spirit of theinvention as set forth in the claims.

What is claimed is:
 1. An eye-tracked head-mounted display, comprising:a micro-display for generating an image to be viewed by a user, themicro-display having a display optical path and an exit pupil associatedtherewith; a first plane located at the micro-display and a second planelocated at the exit pupil; an image sensor configured to receivereflected optical radiation from the exit pupil reflected from a user'seye positioned thereat, the image sensor having a sensor optical pathassociated therewith; display optics disposed in optical communicationwith the micro-display along the display optical path and in opticalcommunication with the image sensor along the sensor optical path, thedisplay optics having a selected surface, closest to the micro-displayand the image sensor for receiving optical radiation from themicro-display; and an illumination source disposed at a locationproximate the micro-display and the selected surface to illuminate theselected surface with optical radiation from the illumination sourcewithout illuminating the micro-display, the display optics configured totransmit the optical radiation from the illumination source and from themicro-display therethrough from the selected surface to the exit pupil.2. The eye-tracked head-mounted display according to claim 1, whereinthe display and image sensor optical paths partially overlap at theselected surface.
 3. The eye-tracked head-mounted display according toclaim 1, wherein the display optics is configured to create a virtualimage of the micro-display for viewing at the exit pupil.
 4. Theeye-tracked head-mounted display according to claim 1, wherein thedisplay optical path and sensor optical path each comprise respectiveoptical axes at the micro-display and image sensor, respectively, andwherein the optical axes are parallel to one another.
 5. The eye-trackedhead-mounted display according to claim 1, wherein the display opticalpath comprises an optical axis, and wherein the image sensor is locatedoff the optical axis of the display optical path.
 6. The eye-trackedhead-mounted display according to claim 1, wherein the display opticalpath comprises an optical axis, and wherein the image sensor is locatedon the optical axis of the display optical path.
 7. The eye-trackedhead-mounted display according to claim 1, comprising a stop at thefirst plane, the stop having at least one aperture therein disposed at alocation along the sensor optical path.
 8. The eye-tracked head-mounteddisplay according to claim 1, comprising a stop having at least oneaperture therein disposed at a location along the sensor optical pathbetween the sensor and selected surface.
 9. The eye-tracked head-mounteddisplay according to claim 8, wherein the at least one aperturecomprises a pin-hole.
 10. The eye-tracked head-mounted display accordingto claim 1, wherein the display optics comprises a freeform opticalelement.
 11. The eye-tracked head-mounted display according to claim 1,wherein the display optics comprises a rotationally symmetric opticalelement.
 12. The eye-tracked head-mounted display according to claim 1,wherein the display optics comprises a freeform optical prism.
 13. Theeye-tracked head-mounted display according to claim 12, wherein theprism comprises a wedge-shaped prism.
 14. The eye-tracked head-mounteddisplay according to claim 12, wherein the prism comprises an asphericsurface.
 15. The eye-tracked head-mounted display according to claim 12,wherein the prism is telecentric in display space.
 16. The eye-trackedhead-mounted display according to claim 12, wherein the prism isnon-telecentric in display space.
 17. The eye-tracked head-mounteddisplay according to claim 12, wherein the prism comprises a TIR (totalinternal reflection) surface oriented to receive and totally internallyreflect light from the micro-display.
 18. The eye-tracked head-mounteddisplay according to claim 12, wherein the prism comprises a TIR (totalinternal reflection) surface oriented to totally internally reflectlight to the image sensor.
 19. The eye-tracked head-mounted displayaccording to claim 12, comprising a freeform corrective lens in opticalcommunication with the prism.
 20. The eye-tracked head-mounted displayaccording to claim 19, wherein field of view of the corrective lens islarger than a field of view of the display optics.